Throughout the history of cyber security, buffer overflow is one of the most serious vulnerabilities in computer systems. (Note that according to this disclosure ‘buffer’ could be a stack or heap.) Buffer overflow vulnerabilities are a “root” cause for most of the cyber attacks such as server break-ins, worms, zombies, and botnets. Buffer overflow attacks are the most popular choice in these attacks, as they provide substantial control over a victim host.
A buffer overflow occurs during program execution when a fixed-size buffer has had too much data copied into it. This causes the data to overwrite into adjacent memory locations, and, depending on what is stored there, the behavior of the program itself might be affected. [2] Taking a broader viewpoint, buffer overflow attacks do not always carry code in their attacking requests (or packets),1 code-injection buffer overflow attacks such as stack smashing may account for most of the buffer overflow attacks that have occurred thus far, 1An attack may direct execution control to existing system code or change the values of certain function arguments.
Although considerable of research has been done to tackle buffer overflow attacks, existing defenses are still quite limited in meeting four highly-desired requirements: (R1) simplicity in maintenance; (R2) transparency to existing (legacy) server OS, application software, and hardware; (R3) resiliency to obfuscation; and (R4) economical Internet wide deployment. As a result, although several very “secure” solutions have been proposed, they are not pervasively deployed, and many buffer overflow attacks continue to succeed each day.
To see how existing defenses are limited in meeting these four requirements, let us break down the existing buffer overflow defenses into six classes which we will review below: (1A) Finding bugs in source code; (1B) Compiler extensions; (1C) OS modifications; (1D) Hardware modifications; (1E) Defense-side obfuscation [3, 4]; and (1F) Capturing code running symptoms of buffer overflow attacks [1, 5-7]. (Note that the above list does not include binary code analysis based defenses which we will address shortly.) We may briefly summarize the limitations of these defenses in terms of the four requirements as follows. (a) Class 1B, 1C, 1D, and 1E defenses may cause substantial changes to existing (legacy) server OS, application software, and hardware, thus they are not transparent. Moreover, Class 1E defenses generally cause processes to be terminated. As a result, many businesses do not view these changes and the process termination overhead as economical deployment. (b) Class 1F defenses can be very secure, but they either suffer from significant runtime overhead or need special auditing or diagnosis facilities which are not commonly available in commercial services. As a result, Class 1F defenses have limited transparency and potential for economical deployment. (c) Class 1A defenses need source code, but source code is unavailable to many legacy applications.
Besides buffer overflow defenses, worm signatures can be generated and used to block buffer overflow attack packets [8-10]. Nevertheless, they are also limited in meeting the four requirements, since they either rely on signatures, which introduces maintenance overhead, or are not very resilient to attack-side obfuscation.